Designing a Situational Judgment Test for Use in Medical School Admissions.

ABSTRACT: It has long been acknowledged that professional competencies are required for success in medical school, residency training, and medical practice. Over the last decade, medical schools have begun to introduce standardized assessments of professional competencies, but many still rely on interviews to assess these competencies, which occur after about half of the applicant pool has already been screened out. In this article, the authors discuss the development, evaluation, and launch of the Association of American Medical Colleges (AAMC) situational judgment test (SJT) for use in medical school admissions. The AAMC SJT is designed to assess an examinee's understanding of effective and ineffective behaviors related to the core competencies for entering medical students, including service orientation, social skills, cultural competence, teamwork, ethical responsibility to self and others, reliability and dependability, resilience and adaptability, and capacity for improvement. The authors evaluate the evidence for the need for SJTs in medical school admissions by exploring common derailers in medical school, gaps in the admissions process regarding information about professional competencies, and the challenge of conducting holistic review in a high-volume context. They summarize existing research from the employment, international medical education, and residency selection contexts suggesting that SJT scores are positively associated with subsequent performance and may add value to the admissions process. The authors discuss 5 goals that were the foundation for developing the AAMC SJT: (1) assess the professional competencies needed for success in medical school using a proven method, (2) enable holistic review in a high-volume admissions context, (3) create and share a program of research to support the appropriate use of SJT scores, (4) signal the need for preparation in professionalism to learners, and (5) balance the need for a new assessment with minimizing the burden and risk for applicants.

Critical Care Teamwork in the Future: The Role of TeamSTEPPS® in the COVID-19 Pandemic and Implications for the Future.

At our institution, we observed inconsistency in the application of structural facilitators for interprofessional teamwork such as handoffs and communication of contingency planning, complete formation and engagement of teams on interprofessional rounds, regular situation monitoring, interprofessional huddles, use of "check back" during code situations, and standard debriefings after codes and procedures (TeamSTEPPS®). To enhance team performance, we piloted TeamSTEPPS® training and reinforcement for all healthcare team members in the medical intensive care unit (MICU), inclusive of trainees, advanced practice providers (APPs), nurses, and respiratory therapists rotating through the unit. Seven months after the training launch, the initial COVID-19 surge interrupted the reinforcement stage of the pilot providing an opportunity to study the retention of TeamSTEPPS® principles and its potential role in response to a crisis. We conducted interprofessional focus groups after a year of crisis management during the pandemic. Themes revealed how TeamSTEPPS® training impacted teamwork and communication, as well as factors that influenced the use of TeamSTEPPS®. This work points to the value of team training in unexpected scenarios. Additional studies at multiple sites are needed to determine scalability for all MICU teams or for onboarding new team members.

Cultivating the physician workforce: Recruiting, training, and retaining physicians to meet the needs of the population.

In the 10 years since the Lancet Commission on Education of Health Professionals for the 21st Century suggested the changes necessary to transform medical education, the United States remains plagued by shortages of physicians and maldistribution of the physician workforce. Minoritized and rural communities usually suffer the most, with widely documented health disparities across the United States by race, ethnicity, gender identity, education, and zip code. Medical schools can respond by recruiting students more likely to practice in these settings and training them to address the community needs. In 2013, the American Medical Association launched an initiative to trigger transformation in medical education and formed a consortium of schools representing a diversity of U.S. institutions. Consortium member schools highlighted in this article share lessons learned in their efforts to strengthen social accountability and develop needed sectors of the physician workforce. Development of the physician workforce involves recruiting and widening pathways of entry for diverse groups, providing training settings and competencies aligned with community needs, and explicit programming in retention, inclusion and well-being to mitigate against workforce losses.

Comparison of self and simulated patient assessments of first-year medical students' Interpersonal and Communication Skills (ICS) during Objective Structured Clinical Examinations (OSCE).

BACKGROUND: Interpersonal and communication skills (ICS) are important core competencies in medical education and certification. In this study, we identified self- and simulated patient (SP)-reported ratings of US first-year medical students' ICS and the influence of age and gender on performance appraisal during the Objective-Structured Clinical Examination (OSCE). METHODS: OSCE participants, including 172 first-year medical students and 15 SPs were asked to evaluate the students' ICS using the American Board of Internal Medicine-Patient-Satisfaction Questionnaire (ABIM-PSQ), electronically and via paper, respectively. Self- and SP-reported ratings of students' ICS were presented as the median on a 5-point Likert-scale and as three categories defined as "good," "very good," and "inadequate." RESULTS: SPs assessed all 172 students in the OSCE, while 43.6% of students assessed their own performance. The majority of students and SPs evaluated the students' ICS as very good. 23.3% of SPs and 5.3% of students rated the medical students' ability to encourage patient question-asking and answer questions as inadequate (P <  0.002). Neither age nor gender influenced the medical students' self-assessment of ICS. Female SPs assigned lower scores to students in regard to respecting patients and encouraging patient question-asking and answering. Older SPs was more likely to assign lower scores on all survey questions. CONCLUSIONS: In the present study, self- and SP-reported ratings of first-year medical students' ICS were mainly "very good" with no influence of students' age or gender. Older age and female gender among the SPs were associated with a reduction in SP-reported ratings of students' ICS.

The Diversity and Success of Medical School Applicants With Scores in the Middle Third of the MCAT Score Scale.

Admissions officers assemble classes of medical students with different backgrounds and experiences who can contribute to their institutions' service, leadership, and research goals. While schools' local interests vary, they share a common goal: meeting the health needs of an increasingly diverse population. Despite the well-known benefits of diversity, the physician workforce does not yet reflect the nation's diversity by socioeconomic status, race/ethnicity, or other background characteristics.The authors reviewed the Medical College Admission Test (MCAT) scores and backgrounds of 2017 applicants, accepted applicants, and matriculants to U.S. MD-granting schools to explore avenues for increasing medical school class diversity. They found that schools that accepted more applicants with midrange MCAT scores had more diverse matriculating classes. Many schools admitting the most applicants with scores in the middle of the MCAT score scale were public, community-based, and primary care-focused institutions; those admitting the fewest of these applicants tended to be research-focused institutions and to report pressure to accept applicants with high MCAT scores to maintain or improve their national rankings.The authors argue that reexamining the use of MCAT scores in admissions provides an opportunity to diversify the physician workforce. Despite evidence that most students with midrange MCAT scores succeed in medical school, there is a tendency to overlook these applicants in favor of those with higher scores. To improve the health of all, the authors call for admitting more students with midrange MCAT scores and studying the learning environments that enable students with a wide range of MCAT scores to thrive.

Development of an empathy and clarity rating scale to measure the effect of medical improv on end-of-first-year OCSE performance: a pilot study.

Patients want empathetic physicians who listen and understand. How do you teach and measure empathy? Medical educators, including those inspired by Alan Alda, have turned to theater to teach skills in empathetic communication. Improvisation-informedcurriculum (medical improv) draws upon foundational actors training: deep listening, emotional understanding, connections, authenticity. Arating scale to measure the impact of medical improv on empathetic and clear communication does not exist. Objective: To develop aframework and instrument, the Empathy and Clarity Rating Scale (ECRS), for measuring communication elements used by actors and physicians, and pilot ECRS to test effectiveness of medical improv on first-yearstudents' communication skills. Design: Four medical schools collaborated. USMLE Step 2 Communication and Interpersonal Skills (CIS) domains were used as framework for discussion among three focus groups, each with clinicians, actors, communication experts, and community members with patient experience. Audiotaped discussions were transcribed; open coding procedures located emerging themes. The initial coding scheme was compared with the Consultation and Relational Empathy (CARE) measure. ECRS content was aligned with CARE, CIS and focus group themes. Modified nominal processes were conducted to finalize the scale. We implemented procedures to establish content validity and interrater reliability. Final ECRS was used to study student performance across three levels of experience with medical improv. Results: The final ECRS was comprised of seven five-pointscale items. Narrative comments precede behaviorally anchored ratings: 5=desired, 1=ineffective, 2-4=developing based upon adjustment needed. Rater agreement across all items was 84%. There was asmall correlation between the ECRS and another measure interviewing (r=0.262, p=0.003). Students with advanced medical improv training outperformed those without (F=3.51, p=.042). Conclusion: Acommunication scale enlightened by experiences of actors, clinicians, scholars and patients has been developed. The ECRS has potential to detect the impact of medical improv on development of empathetic and clear communication.

Machine Learning-Augmented Propensity Score-Adjusted Multilevel Mixed Effects Panel Analysis of Hands-On Cooking and Nutrition Education versus Traditional Curriculum for Medical Students as Preventive Cardiology: Multisite Cohort Study of 3,248 Trainees over 5 Years.

BACKGROUND: Cardiovascular disease (CVD) annually claims more lives and costs more dollars than any other disease globally amid widening health disparities, despite the known significant reductions in this burden by low cost dietary changes. The world's first medical school-based teaching kitchen therefore launched CHOP-Medical Students as the largest known multisite cohort study of hands-on cooking and nutrition education versus traditional curriculum for medical students. METHODS: This analysis provides a novel integration of artificial intelligence-based machine learning (ML) with causal inference statistics. 43 ML automated algorithms were tested, with the top performer compared to triply robust propensity score-adjusted multilevel mixed effects regression panel analysis of longitudinal data. Inverse-variance weighted fixed effects meta-analysis pooled the individual estimates for competencies. RESULTS: 3,248 unique medical trainees met study criteria from 20 medical schools nationally from August 1, 2012, to June 26, 2017, generating 4,026 completed validated surveys. ML analysis produced similar results to the causal inference statistics based on root mean squared error and accuracy. Hands-on cooking and nutrition education compared to traditional medical school curriculum significantly improved student competencies (OR 2.14, 95% CI 2.00-2.28, p < 0.001) and MedDiet adherence (OR 1.40, 95% CI 1.07-1.84, p = 0.015), while reducing trainees' soft drink consumption (OR 0.56, 95% CI 0.37-0.85, p = 0.007). Overall improved competencies were demonstrated from the initial study site through the scale-up of the intervention to 10 sites nationally (p < 0.001). DISCUSSION: This study provides the first machine learning-augmented causal inference analysis of a multisite cohort showing hands-on cooking and nutrition education for medical trainees improves their competencies counseling patients on nutrition, while improving students' own diets. This study suggests that the public health and medical sectors can unite population health management and precision medicine for a sustainable model of next-generation health systems providing effective, equitable, accessible care beginning with reversing the CVD epidemic.

How good are we at selecting students that meet our mission? Outcomes of the 2011 and 2012 entering classes selected by a locally developed multiple mini interview.

BACKGROUND: Can a locally developed multiple mini interview (MMI) process lead to outcomes reflective of local values and mission? METHODS: In 2017, the authors performed a retrospective analysis of the relationship of MMI with multiple-choice-based outcomes and non-multiple-choice-based outcomes, including clerkship competencies, OSCE, scholarship/service/leadership, academic honor society induction, peer and faculty humanism nominations, and overall performance at graduation for two entering classes with acceptance decisions based exclusively on a locally developed MMI. RESULTS: There was no association between MMI and performance on multiple-choice-based examinations. For other outcomes, the effect size of MMI for OSCE was small and leadership/service and scholarship did not correlate with MMI score. For clerkship competencies, there was medium effect size for patient care, practice-based learning and improvement, interpersonal and communication skills, and cultural competence. Highest and lowest quartile MMI scorers were no different in academic honor society induction; however, top quartile MMI scorers received more humanism votes versus last quartile and were more likely rated outstanding or excellent graduates. CONCLUSIONS: Local development of MMI and of admissions processes with sole reliance on MMI for final acceptance decisions will not affect academic preparation/medical school performance in multiple-choice-based assessments but can lead to locally desired attributes in students.

Implementation of a Mini-CEX Requirement Across All Third-Year Clerkships.

PROBLEM: Direct observation of medical students performing clinical tasks, such as eliciting a patient history or examining a patient, and the provision of feedback, are foundational to student improvement but have been reported to occur infrequently. The mini clinical evaluation exercise (mini-CEX) is a tool that can facilitate direct observation and feedback. This study assessed the impact of a mini-CEX requirement across all 3rd-year clerkships on student report of direct observation by faculty and objectively measured clinical skills. INTERVENTION: A mini-CEX requirement across all 3rd-year clerkships was implemented in the 2012-2013 academic year. The impact of the mini-CEX requirement on student report of direct observation was assessed by end-of-clerkship surveys and Association of American Medical Colleges (AAMC) Graduation Questionnaire (GQ) items on direct observation. The impact on students' clinical skills was assessed by a summative Objective Structured Clinical Examination (OSCE). Pre/post comparisons were assessed with chi-square and Fisher's exact tests. CONTEXT: A mini-CEX requirement had been in place for the internal medicine clerkship, and student reports of direct observation were historically higher for the internal medicine clerkship than for other clerkships. Faculty, residents, and students at each of the clinical sites across all 6 clerkships were oriented to the use of the mini-CEX; the feasibility of its use during usual patient interaction settings and the importance of direct observation and feedback for student improvement were emphasized during these sessions. OUTCOME: Adherence to the mini-CEX requirement was high: 92% of required forms were completed, and 78% of completed forms indicated that specific feedback was given. The proportion of students reporting direct observation of physical examination significantly increased in all clerkships, with the largest relative increase occurring in surgery (from 49% to 87%), χ2(1, N = 225) = 37.70, p < .0001. Significant increases were seen in faculty observation of history taking in pediatrics, surgery, and psychiatry. Direct observation rates also increased on the AAMC GQ items for history taking and physical exam for all clerkships. Failures on the summative OSCE decreased from 12% preintervention to 2% postintervention (p = .0046). LESSONS LEARNED: Institution of a mini-CEX requirement was feasible across all 3rd-year clerkships and was associated with a significant increase in student report of direct observation by faculty and a decrease in summative OSCE failure rates.

The Effect of Differential Weighting of Academics, Experiences, and Competencies Measured by Multiple Mini Interview (MMI) on Race and Ethnicity of Cohorts Accepted to One Medical School.

PURPOSE: To examine whether academic scores, experience scores, and Multiple Mini Interview (MMI) core personal competencies scores vary across applicants' self-reported ethnicities, and whether changes in weighting of scores would alter the proportion of ethnicities underrepresented in medicine (URIM) in the entering class composition. METHOD: This study analyzed retrospective data from 1,339 applicants to the Rutgers Robert Wood Johnson Medical School interviewed for entering classes 2011-2013. Data analyzed included two academic scores-grade point average (GPA) and Medical College Admission Test (MCAT)-service/clinical/research (SCR) scores, and MMI scores. Independent-samples t tests evaluated whether URIM ethnicities differed from non-URIM across GPA, MCAT, SCR, and MMI scores. A series of "what-if" analyses were conducted to determine whether alternative weighting methods would have changed final admissions decisions and entering class composition. RESULTS: URIM applicants had significantly lower GPAs (P < .001), MCATs (P < .001), and SCR scores (P < .001). However, this pattern was not found with MMI score (non-URIM 10.4 [1.6], URIM 10.4 [1.3], P = .55). Alternative weighting analyses show that including academic/experiential scores impacts the percentage of URIM acceptances. URIM acceptance rate declined from 57% (100% MMI) to 43% (10% GPA/10% MCAT/10% SCR/70% MMI), 39% (30% GPA/70% MMI), to as low as 22% (50% MCAT/50% MMI). CONCLUSIONS: Sole reliance on the MMI for final admissions decisions, after threshold academic/experiential preparation are met, promotes diversity with the accepted applicant pool; weighting of "the numbers" or what is written about the application may decrease the acceptance of URIM applicants.

Core personal competencies important to entering students' success in medical school: what are they and how could they be assessed early in the admission process?

Assessing applicants' personal competencies in the admission process has proven difficult because there is not an agreed-on set of personal competencies for entering medical students. In addition, there are questions about the measurement properties and costs of currently available assessment tools. The Association of American Medical College's Innovation Lab Working Group (ILWG) and Admissions Initiative therefore engaged in a multistep, multiyear process to identify personal competencies important to entering students' success in medical school as well as ways to measure them early in the admission process. To identify core personal competencies, they conducted literature reviews, surveyed U.S and Canadian medical school admission officers, and solicited input from the admission community. To identify tools with the potential to provide data in time for pre-interview screening, they reviewed the higher education and employment literature and evaluated tools' psychometric properties, group differences, risk of coaching/faking, likely applicant and admission officer reactions, costs, and scalability. This process resulted in a list of nine core personal competencies rated by stakeholders as very or extremely important for entering medical students: ethical responsibility to self and others; reliability and dependability; service orientation; social skills; capacity for improvement; resilience and adaptability; cultural competence; oral communication; and teamwork. The ILWG's research suggests that some tools hold promise for assessing personal competencies, but the authors caution that none are perfect for all situations. They recommend that multiple tools be used to evaluate information about applicants' personal competencies in deciding whom to interview.

Creative group performances to assess core competencies in a first-year patient-centered medicine course.

INTRODUCTION: A novel assessment of systems-based practice and practice-based learning and improvement learning objectives, implemented in a first-year patient-centered medicine course, is qualitatively described. METHODS: Student learning communities were asked to creatively demonstrate a problem and solution for health care delivery. Skits, filmed performances, plays, and documentaries were chosen by the students. Video recordings were reviewed for themes and the presence of course competencies. RESULTS: All performances demonstrated not only the index competencies of team work and facilitation of the learning of others, but many other core objectives of the course. The assignment was rated positively both by the faculty and the students, and has been added to the assessment modalities of the course.

Inflammatory cytokines provide limited early prognostic information in emergency department patients with suspected myocardial ischemia.

STUDY OBJECTIVE: The aim of this study was to assess the early prognostic value of the inflammatory cytokines interleukin 6, interleukin 8, and tumor necrosis factor alpha in a cohort of emergency department (ED) patients with chest pain who have suspected myocardial ischemia. METHODS: One hundred eighteen patients with chest pain presenting to 2 urban EDs were studied. Interleukin 6, interleukin 8, and tumor necrosis factor alpha levels were assayed at presentation. The end point was the occurrence of a serious cardiac event (death, nonfatal acute myocardial infarction, myocardial revascularization, or readmission with an acute coronary syndrome) during the index admission or subsequent 3 months. RESULTS: Mean levels of all 3 cytokines were higher among patients experiencing a serious cardiac event, with the greatest differences observed in levels of interleukin 6 (mean 2.5 pg/mL [95% confidence interval (CI) 1.2 to 3.7 pg/mL] versus mean 9.8 pg/mL [95% CI 2.4 to 17.2 pg/mL]). Interleukin 6 had a sensitivity of 35% (95% CI 20% to 54%), a specificity of 86% (95% CI 76% to 92%), and an overall prognostic accuracy of 71% (95% CI 63% to 79%) for predicting serious cardiac events. However, logistic regression analysis revealed that the only independent predictor of an adverse outcome was an ECG suggestive of ischemia at presentation. CONCLUSION: Among patients presenting to the ED with suspected myocardial ischemia, higher levels of inflammatory cytokines are associated with an increased risk of a serious cardiac event during the subsequent 3 months. There is, however, considerable overlap in levels among patients who do and do not have a serious cardiac event, limiting their utility as predictors of outcome in individual patients.